Oxford Trecvid 2006 – Notebook Paper 1 High-level Feature Extraction 1.1 Bag of Visual Word Representation

The Oxford team participated in the high-level feature extraction and interactive search tasks. A vision only approach was used for both tasks, with no use of the text or audio information. For the high-level feature extraction task, we used two different approaches, one using sparse and one using dense visual features to learn classifiers for all 39 required concepts, using the training data supplied by MediaMill [29] for the 2005 data. In addition, we also used a face specific classifier, with features computed for specific facial parts, to facilitate answering people-dependent queries such as “government leader”. We submitted 3 different runs for this task. OXVGG_A was the result of using the dense visual features only. OXVGG_OJ was the result of using the sparse visual features for all the concepts, except for “government leader”, “face” and “person”, where we prepended the results from the face classifier. OXVGG_AOJ was a run where we applied rank fusion to merge the outputs from the sparse and dense methods with weightings tuned to the training data, and also prepended the face results for “face”, “person” and “government leader”. In general, the sparse features tended to perform best on the more object based concepts, such as “US flag”, while the dense features performed slightly better on more scene based concepts, such as “military”. Overall, the fused run did the best with a Mean Average (inferred) Precision (MAP) of 0.093, the sparse run came second with a MAP of 0.080, followed by the dense run with a MAP of 0.053. For the interactive search task, we coupled the results generated during the high-level task with methods to facilitate efficient and productive interactive search. Our system allowed for several “expansion” methods based on the sparse and dense features, as well as a novel on the fly face classification system, which coupled a Google Images search with rapid Support Vector Machine (SVM) training and testing to return results containing a particular person within a few minutes. We submitted just one run, OXVGG_TVI, which performed well, winning two categories and coming above the median in 18 out of 24 queries. 1 High-level Feature Extraction Our approach here is to train an SVM for the concept in question, then score all key frames in the test set by the magnitude of their discriminant (the distance from the discriminating hyper-plane), and subsequently rank the test shots by the score of their keyframes. We have developed three methods for this task, each differing in their features and/or kernel. Two of the methods are applicable to general visual categories (such as airplane, mountain and road) and the third is specific to faces. The first two methods differ in that one uses sparse (based on region detectors) monochrome features, and the other uses dense (on a regular pixel grid) colour features. We now describe the three methods in some detail. 1.1 Bag of visual word representation The first approach uses a bag of (visual) words [27] representation for the frames, where positional relationships between features are ignored. This representation has proved successful for classifying images according to whether they contain visual categories (such as cars, horses, etc) by training an SVM [7]. Here we use the kernel formulation proposed by [31]. Features and bag of words representation: We used two types of affine region detectors, Hessian Laplace (HL) [21], and Maximally Stable Extremal Regions (MSER) [20]. Each region is then represented by a SIFT [19] descriptor using intensity only. This combination of detection and description generates features which are approximately invariant to an affine transformation of the image, see figure 1. These features are computed in all (representative and nonrepresentative) keyframes. The ‘visual vocabulary’ is then constructed by vector quantizing the SIFT descriptors of a random subset of features from the training data using Kmeans. The K-means cluster centres define the visual words. Each feature type (HL and MSER) has its own vocabulary. We used two vocabulary sizes of 3,000 and 10,000 words for HL, and one vocabulary size of 3,000 words for MSER, ref-